Gesture recognition system using depth perceptive sensors

ABSTRACT

Acquired three-dimensional positional information is used to identify user created gesture(s), which gesture(s) are classified to determine appropriate input(s) to an associated electronic device or devices. Preferably at at least one instance of a time interval, the posture of a portion of a user is recognized, based at least one factor such as shape, position, orientation, velocity. Posture over each of the instance(s) is recognized as a combined gesture. Because acquired information is three-dimensional, two gestures may occur simultaneously.

PRIORITY INFORMATION

Priority is claimed from co-pending U.S. patent application Ser. No.10/369,999 filed 18 Feb. 2003 entitled “Gesture Recognition System UsingDepth Perceptive Sensors”, soon to issue as U.S. Pat. No. 7,340,777, andassigned to assignee herein, Canesta, Inc. of Sunnyvale, Calif. The '999application claimed priority from U.S. provisional patent applicationNo. 60/357,730, entitled “Natural Touch and Gesture Based Interactionfor Electronic Systems,” filed Feb. 15, 2002, from U.S. provisionalpatent application No. 60/394,068, entitled “Shape Representation andRecognition by Random Histograms”, filed Jul. 2, 2002, and from U.S.provisional patent application No. 60/410,415, entitled “GestureRecognition System with 3D Input,” filed on Sep. 13, 2002. Each of theseprovisional patent applications was assigned to Canesta, Inc.

FIELD OF THE INVENTION

The present invention relates generally to electronic perceptiontechnology, and more particularly to a gesture recognition system thatuses depth perceptive sensors.

BACKGROUND OF THE INVENTION

Electronic systems exist for using gestures, such as those created bythe movement of a hand, as input. For example, there are handwritingrecognition devices that interpret a user's gesture made through astylus or pen as input. Also, there are systems that gear (or provide)users with wiring or other implements in order to track the user's handor body movements using optical sensors. For example, U.S. Pat. No.6,252,598, describes the use of video images to identify hand gestures.A plurality of regions in the frame are defined and screened to locatean image of a hand in one of the regions. A hand image is processed tolocate extreme curvature values, such as peaks and valleys,corresponding to predetermined hand positions and gestures. The numberof peaks and valleys are then used to identify and correlate apredetermined hand gesture to the hand image for effectuating aparticular computer operation or function. In order to find thecurvature values on the hand, the boundaries of the hand must bereliably obtained. This can be problematic because the edges of anintensity image are closely related to the lighting and backgroundproperties of the scene. Furthermore, the intensity of the image makesuse of the system dependent on the lighting of the scene.

U.S. Pat. Nos. 6,256,033 and 6,072,494 provide for acomputer-implemented gesture recognition system. These systems require abackground image model to be created by examining frames of an averagebackground image before the subject that will perform the gesture entersthe image. The necessity of having the background picture reduces thepractical applicability of the method, particularly since the backgroundcan change due to movements in the background, or changes to lighting orshadows.

U.S. Pat. No. 6,222,465 describes a system and method for manipulatingvirtual objects in a virtual environment, for drawing curves and ribbonsin the virtual environment, and for selecting and executing commands forcreating, deleting, moving, changing, and resizing virtual objects inthe virtual environment using intuitive hand gestures and motions. Thesystem is provided with a display for displaying the virtual environmentand conceptual description of a video gesture recognition subsystem foridentifying motions and gestures of a user's hand.

U.S. Pat. No. 6,204,852 describes a video gesture-basedthree-dimensional computer interface system that uses images of handgestures to control a computer. The system tracks motion of the user'shand or an elongated object or a portion thereof in a three-dimensionalcoordinate system with five degrees of freedom. The system containsmultiple cameras. These cameras are not used to obtain a depth image ofthe scene. Instead, every camera image is processed independently, andthe location of the finger is located in both of the cameras. Thelocation of the finger is next located by the geometry between thecamera's and the finger's location in each image. The orientation of thefinger is determined in a similar manner. The method is intended to beused for applications with a pointing finger. Furthermore, if multiplefingers are used in the gesture, the method may not be able tounambiguously determine the corresponding fingers in each image.

U.S. Pat. No. 5,781,663 describes an image recognition apparatus thatoperates in three modes. A gesture recognition mode is used to recognizean input locus as a command, a figure recognition mode is used torecognize a figure, and a character recognition mode is used torecognize a character.

U.S. Pat. Nos. 5,454,043, 6,002,808, and 5,594,469, each provide agesture recognition framework using intensity images. The patentsillustrate the use of moments and frequency histograms for imagerepresentation and recognition. The algorithms described therein rely onthe edges of the hands in the intensity images, and therefore thedescribed systems are very much affected by ambient conditions of theenvironment, such as by a background that has similar color with theskin. For instance, the system might misinterpret the edges when thereis not enough illumination on the foreground (hand) so that edgesbetween the foreground and background disappear.

SUMMARY OF THE INVENTION

In an embodiment, a method is provided that enables a person (e.g.,user) to interact with an electronic device. The method includesobtaining position information for a plurality of discrete regions on abody part of the person. The position information indicates a depth ofeach discrete region on the body part relative to a reference. Thegesture is classified using the position information. The classificationof the gesture may be used as input for interacting with the electronicdevice.

Other features and advantages of the invention will appear from thefollowing description in which the preferred embodiments have been setforth in detail, in conjunction with the accompany drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example, and notby way of limitation, in the figures of the accompanying drawings. Likereference numerals are intended to refer to similar elements amongdifferent figures.

FIG. 1 illustrates a gesture recognition system for using depthperception to recognize the gesture created by a body part of a person;

FIG. 2 illustrates a method for implementing a gesture recognitionsystem, according to an embodiment of the invention;

FIG. 3 describes a method for using depth information to segment a bodypart from an image of a scene so that the gesture made with the bodypart can be identified and classified;

FIG. 4 illustrates a method for recognizing a posture of a body part ofinterest, under an embodiment of the invention;

FIGS. 5A-5I illustrate a series of snap shots that show the use ofhistograms to identify hand gestures;

FIG. 6 provides a more detailed description of a method for determininga pose of a body part of interest, where the body part of interest isassumed to be a hand;

FIG. 7 illustrates another embodiment where the beginning and end of adynamic gesture are determined through analysis of discrete bodygestures recognized at consecutive instances of time.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the invention describe methods and apparatuses forrecognizing gestures of body parts using depth perceptive sensors. Inthe following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

A. Overview

Embodiments of the invention provide for techniques that can be employedto identify and/or recognize the movement or posture of objects in athree-dimensional space. The identification or recognition of theobject's movement may be used as input for an electronic device orsystem, including, for example, appliance, a mobile device such as acell phone or PDA, a game console, an audio/visual entertainment center,a toy, or a robot. The electronic device may assign a meaning to thatinput and take an action specific to that system. In one embodiment,algorithms for classifying gestures as input are performed within theelectronic system (input is generated internally) where the input is tobe entered. In another embodiment, such algorithms are performed on aseparate computer system that communicates with the electronic devicewhere the input is to be received over a communication line. Aparticular point of interest is the identification and recognition of aperson's body movements, with specific interest in hand gestures.However, embodiments of the invention may pertain to all body movements,such as arm or foot movements, in addition to movements by other typesof objects, including objects such as stylus, pointers and other objectsthat are carried by persons.

Embodiments of the invention also include components or processes foruse in identifying and/or recognizing the movement or posture of objectsin three-dimensional space. Such components or processes may provide forobtaining depth information from an object that is making the gesture, asegmentation module to separate an image of the object from thebackground, a posture representation module that identifies a specificpose and/or shape of the object, a gesture representation module thatidentifies the overall gesture and a classification module thatclassifies the recognized gesture as a designated input.

In one embodiment, a three-dimensional gesture recognition system iscommunicatively coupled to a translation system in order to translategesture recognition information into one or more specific inputs.According to embodiments described herein, a system is provided todetect the relative position of a user's fingers, hands, or other bodypart, and to interpret these movements, precisely as they are made, intoan electronic system as input. A system such as described accomplishesthis without requiring the user to be outfitted with wires, markers orother physical implements.

In another embodiment, a system or method is provided to detectpositioning and movement of a user's finger, hand or body in real-time.The system may, for example, provide feedback to the user about theuser's movements and position.

In contrast to existing systems, embodiments of the invention enable auser to utilize his own fingers, hands, or other body movements, withoutany physical implements for tracking movement, in order to control theelectronic system.

Embodiments of the invention provide several advantages over otherapproaches that have been developed for recognizing gestures of bodyparts and other movements. For example, embodiments described herein usedepth information, which is independent of lighting conditions in theenvironment. This is in contrast to other approaches that rely onanalysis of light intensity. Light intensity is often too sensitive tobe reliable. Techniques described for use with embodiments of theinvention eliminate the background variables by using depth images. Theresult is a gesture recognition system that is robust to changes inlighting or appearance of the background.

In addition, embodiments described herein are suitable for recognizing avariety of types of gestures, including hand motions, and motions ofvarying intensity and speed. In contrast, many past approaches have beenlimited to pointing and handwriting applications, which requirerelatively simple gestures in a very limited environment (e.g. only on acomputer tablet).

Embodiments of the invention may be implemented in the form ofinstructions that are executable by one or more processors. Theinstructions may then be provided, stored, or carried, on acomputer-readable medium. Examples of computer-readable memory includesmemory or a communication channel or medium.

B. Terminology

The term “body gesture” or “hand gesture” refers to a set of one or moreconsecutive set of body or hand postures. A body/hand gesture is staticif only one posture is identified or recognized. A body/hand gesture isdynamic if more than one consecutive hand postures are identified orrecognized collectively. Such consecutive hand postures may result fromcapturing the body part moving between a starting position and an endingposition, or over a series of time intervals. (If the user holds astylus, a wand, etc., it is understood that such user-held device willbe included in the term “body gesture” or “hand gesture”.)

The term “image” means an instance of light recorded on a tangiblemedium. The image does not have to be a recreation of the reflection,but merely record a characteristic such as brightness, particularly fromvarious points of a surface or area in which a reflection is beingcreated. The tangible medium may refer to, for example, an array oflight-sensitive pixels.

As used herein, a “module” includes logic, a program, a subroutine, aportion of a program, a software component or a hardware componentcapable of performing a stated task, function, operation, or process. Amodule can exist as hardware, software, firmware, or combinationthereof. Furthermore, one module may be distributed over severalcomponents or physical devices, so long as there are resources thatcooperate with one another to perform the functions of the module.

The term “depth” means a depth-wise distance. The depth refers to adistance between a sensor and an object that is being viewed by thesensor. The depth can also be a relative term such as the verticaldistance from a fixed point in the scene closest to the camera.

A “computer-readable medium” includes any medium wherein stored orcarried instructions can be retrieved or otherwise read by a processorthat can execute the instructions.

C. Gesture Recognition System and Method

A gesture recognition system is described for detecting and recognizingbody gestures. The use of hand gestures is only one type of object thatcan be used with embodiments of the invention. The hand, in particular,may have various complex movements and postures, reflecting positioningand orientation of individual fingers, the wrist and/or the palm. Thus,the description provided for detecting and recognizing hand gestures isillustrative of the capabilities provided by embodiments of theinvention.

FIG. 1 illustrates a gesture recognition system 100 for using depthperception to recognize the gesture created by a body part of a person,under an embodiment. The body part that is recognized by a system suchas described in FIG. 1 may include, for example, a finger, a hand, anarm, a leg, a head, a shoulder or a combination thereof. The body partmay include other moveable parts of a person, such as the eye lidswinking or the eye balls moving side to side, and the mouth opening ormoving (such as when the person is speaking). The hand gesturerecognition system includes a three-dimensional sensor system 110, asegmentation module 120, a body posture module 130, and a classificationmodule 140. In addition, a body gesture representation module 150 may beprovided as one of the optional features for the gesture recognitionsystem 100. T he gesture recognition system 100 may operate over aregion 155.

A system such as described in FIG. 1 is invariant to the type of threedimensional sensor system that is used. One suitable three-dimensionalsensor system is described in U.S. Pat. No. 6,323,942 (herebyincorporated by reference in its entirety for all purposes). This patentdescribes a system that sends out a pulse of light to a targeted region,and measures the time that the pulse takes to travel. Another type ofthree-dimensional sensor is a stereo camera method. This type of sensorsystem uses two cameras and finds the location of every correspondingpoint in both images. An example of such a sensor system is described inliterature such as Machine Vision, by Rmnesh Jain, Rangachar Kasturi,Brian G. Schunck, published by McGraw-Hill in 1995.

For purpose of description, a person 162 located within the region 155is able to move or position a particular body part, such as a hand 165,in order to enter input for a related electronic device 105. In oneembodiment, the gesture recognition system 100 may be provided as anintegral part of the electronic device 105. The electronic device 105includes virtually any type of device, including a portable computer(such as a personal digital assistant or laptop), smart cellular phone,television set, stereo system, musical instrument, robot, toy, gamingconsole, or appliance. The gesture recognition system 100 may recognizegestures of the person 162 to provide input to the electronic device105. In another embodiment, the gesture recognition system 100 may beseparate from the electronic device 105, but coupled to the electronicdevice through a communication medium. The communication medium may bewireless. For example, the communication medium may be infrared or aradio-frequency channel. The communication medium may also be a physicalconnection, such as a serial bus line. The gesture recognition system100 may be applicable to various environments. For example, the gesturerecognition system 100 may be employed in a car (adjustment of carstereo), in a home (control television and audio system), or in abusiness area (control electronic doors).

The three-dimensional sensor system 110 may identify positioninformation for the body part, including depth-wise position informationfor discrete portions of the object. The three-dimensional sensor system110 may create a depth image that contains position information of theentire scene in the region 155, including position information of thebody part of interest. The segmentation module 120 may segment the bodypart of interest from background and other objects in the depth image.The body posture module 130 determines the shape and position of thebody part of interest statically at one particular interval. The dynamicdetermination may be determined when the body part moves in the region155. The gesture representation module 150 is used to identify thegesture created dynamically over a duration by the body part if movementof the body part is to be a factor in determining what the input fromthe user is.

The identified body gesture may be classified by classification module140. The classification module 140 may classify the identified bodygesture to a particular body gesture among a set of multiple bodygestures. The particular body gesture may then be correlated into input.

FIG. 2 illustrates a method for implementing a gesture recognitionsystem, according to one embodiment. A method such as described in FIG.2 may be implemented using components such as described by FIG. 1.

Step 210 provides that a depth image is obtained from a scene where ahand gesture or other body movement is made. The depth image may beobtained using a three dimensional sensor system that can determine arange between discrete portions of an object and a sensor system. Insome applications, a three-dimensional sensor is sometimes referred toas a “2½ dimensional” sensor system. In either case, an embodiment suchas described by FIG. 2 provides that the sensor system obtainsdepth-wise position information for discrete portions of an object thathas or is making the gesture.

A sensor system for use with such an embodiment may employ various threedimensional sensing techniques. For example, the sensor system mayreflect and modulate light and use the time of flight to determine therange of discrete portions of an object. Other embodiment may use one ormore techniques, including active triangulation, stereovision, depthfrom de-focus, structured illumination, and depth from motion. U.S. Pat.No. 6,323,942, entitled “CMOS Compatible 3-D Image Sensor” (herebyincorporated for all purposes in its entirety) describes components andtechniques that can be employed to obtain the sensor information.

The sensor systems may provide a three-dimensional depth image of thescene. The depth image may be captured on a panel of pixels, where eachpixel contains a discrete portion of the image attributable from oneportion of the surface being imaged. Each pixel that forms the depthimage gives an indication of depth information corresponding to adistance between the portion of the surface being imaged by that pixeland the sensor system. If, for example, three-dimensional sensor systemsuch as described in U.S. Pat. No. 6,323,942 is used, distanceinformation is collected for the space represented by each pixel in afield of the sensor system's view.

In step 220, the depth image is analyzed in order to segment the hand orother body component of interest from other objects and backgroundcaptured by the depth image. A method such as described in FIG. 3 may beused to segment the hand or other body component of interest.

Step 230 corresponds to detecting when the gesture starts. According toone embodiment, the beginning and end of the gesture are identifiedusing delimiters that are separate from analysis of the hand gesture.For example, the user may perform a designated action, such as creatinga specific sound (clapping) or creating a specific hand posture, inorder to signal the beginning of the hand gesture. In another embodimentsuch as described with FIG. 7, the analysis of the hand gesture alsoidentified the start and end of the hand gesture.

From when the hand gesture starts, one or more intervals are used inorder to recognize the hand gesture as an input. Each intervalcorresponds to the time where a hand posture is determined. The handposture is a hand gesture that is viewed statically in a particularinterval of time. Step 240 provides that the shape of a hand at thecurrent time interval is determined. FIG. 4 illustrates one method,under an embodiment of the invention, for determining the shape of ahand when the hand is making a gesture.

In step 250, the pose of the hand is identified. The pose of the handrefers to the position of the hand in three coordinates within theregion of observation, and the orientation of the hand in that region.FIG. 5 describes one method, under an embodiment of the invention, fordetermining the pose of a hand when the hand is making a gesture.

In step 260, the shape and pose of the hand at the particular intervalis recorded. For example, the shape and pose may be stored in memory,and made available for combination with other determinations of theshape and pose of the hand in preceding or succeeding intervals.

In step 265, a determination is made as to whether the gesture hasended. As with step 230, this determination may be used using delimitersthat are separate from analysis of the hand gesture. Alternatively, theend of the gesture may be determined by analyzing hand postures insucceeding intervals to determine when the hand gesture has started andstopped. If the hand gesture has not stopped, steps 240-265 are repeatedfor the next time interval.

If the hand gesture has stopped, step 270 provides that the gesture isidentified. If the hand gesture is a posture (one interval), the handgesture may be identified by determining the shape and pose of the hand(see FIG. 4 and FIG. 5). Otherwise, a method such as described in FIG. 7may be used to identify the hand gesture when the hand gesture hasvarying or dynamic hand gestures, or when the hand gesture lasts for aduration of two or more intervals.

In step 280, the hand gesture is classified. For example, the identifiedhand gesture may be compared to a set of multiple designated handgestures. The comparison may select one classification for theidentified hand gesture. This may correspond to correlating the handgesture to input data. The input data can be a simple code or a seriesof commands to an electronic system.

D. Segmentation

Segmentation refers to a process where a body part of interest isdelineated from other objects in a scene that is being observed forpurpose of recognizing a gesture made by that body part. FIG. 3describes a method for using depth information to segment a body partfrom an image of scene in order to identify and classify the gesturemade with that body part. For purpose of description, the body part maybe assumed to be a hand that gestures. A method such as described inFIG. 3 may be performed by the segmentation module 120 in FIG. 1.Reference to elements of FIG. 1 is for descriptive purposes only.

A method such as described in FIG. 3 uses the depth informationcontained within the depth image to segment the hand (or other body partof interest) for the gesture recognition system. The depth imagecontains a depth value at each of its pixels. Step 310 provides that thepixel values for the depth image of the scene are identified. The depthvalue of each pixel corresponds to the depth of a particular discreteportion of the visible region that can be sensed with the sensor system110. For example, the depth image of a scene that contains one or morehands in front of the background will have pixels that map to hands andpixels that map to the other objects in the background.

In step 320, adjacent pixels that have similar depth values are groupedtogether. If there is a prior expectation for the depth of a hand, thenthe objects that are inconsistent with that expectation can be directlyeliminated. In order to group pixels with similar depths, standardsegmentation algorithms could be applied on the remainder of the depthimage. For instance, the classical image split-and-merge segmentationmethod by Horowitz and Pavlidis provides for splitting the image intoparts. It then tests both individual and adjacent parts for“homogeneity” according to some user-supplied criterion. If a singlepart does not satisfy the homogeneity criterion, it is split into twoparts. If two adjacent parts satisfy the criterion even after they aretentatively regarded as a single region, then the two parts are merged.The algorithm continues this procedure until no region need be split,and no two adjacent regions can be merged. Although this algorithm wasdesigned for regular brightness or color images, it could be applied todepth images as well.

An alternative is to use this or another segmentation algorithm on thegradient of the depth image, so that the value of any threshold used inthe homogeneity criterion becomes less critical. Specifically, a regioncan be declared to be homogeneous when the greatest gradient magnitudein its interior is below a predefined threshold.

Still further, another alternative is to use the k-means algorithm tocluster the depth map into regions with similar depths. However, oneshortcoming of such algorithms is that it is usually hard to determine apriori a good value for the number k of clusters to be computed. Toovercome this problem, an adaptive scheme for the selection of k can beapplied. Standard image segmentation methods such as a normalized cutmethod can also be applied to find the segments that belong to objectsat different depths.

Step 330 provides that pixel groups that correspond to a portion of ahand are identified. In one embodiment, pixel groups of a particulardepth or assumed to be portions of the hand. For example, the sensorsystem 110 may be aligned so that the hand of the person is the closestobject to a camera of the sensor system. The pixel groups that havedepth information identifying them as the closest group may beidentified as portions of the hand.

In an other embodiment, a camera or other image capturing device,including one that is used with the sensor system 110, may capturelight-intensity information with a second panel of pixels. In such anembodiment, intensity characteristics that identify shading orpigmentation may be analyzed to match predetermined values correspondingto the skin on a person's hand.

E. Posture Recognition and Representation

Embodiments of the invention provide for recognizing and representingthe posture of body parts of interest, and in particular hand gestures.FIG. 4 illustrates a method for recognizing a posture of a body part ofinterest, under an embodiment of the invention. The body posture may bedetermined at one instance, or at many instances over a given duration,in order to correlate the posture as an input for a related electronicdevice. For purpose of description, the body part is assumed to be ahand that gestures. A method such as described in FIG. 4 may beperformed by the body posture module 130 in FIG. 1. Reference toelements of FIG. 1 is for descriptive purposes only.

In order to determine body posture, the shape of the body part ofinterest, and the pose of the part are determined. The pose of the bodypart refers to its position and the orientation.

According to one embodiment, a statistical methodology is used todetermine the shape of the body part. One particularly usefulstatistical methodology is the histogram, which can represent images ina compressed form. Random histogram information can be used toaccurately represent an image by collecting statistics on a sufficientlylarge sample of pixels.

In an embodiment, a pose of the body part may be determined using amethodology such as singular value decomposition. The pose, incombination with the shape of the body part, provide the body partposture. As will be described, the body part posture, or a gesture thatcomprises the body part posture, are identifiable as input to anelectronic device.

With reference to FIG. 4, step 410 provides that sampling parameters aredefined for a histogram that is to be created from the segmented imageof the body part of interest (determined with a method described in FIG.3). Histograms are based on identifying the number of occurrences of adata point that has a particular aspect. The idea is simple to implementwhen data points contain one-dimensional data. An example ofone-dimensional information is the distance between random points. Forthis case, histogram can directly be obtained by counting the number ofoccurrences towards each bin.

The sampling parameters define the aspect of the segmented image that isto be the source for information contained in a histogram. Once properlyconstructed, such a histogram can be used for determining the shape ofthe body part. Specifically, the sampling parameters define what is tobe a bin (aspect) of the histogram, and what is to be contained in thebin (histogram value). In particular the aspect of the segmented imagecorresponds to discrete locations of the body part of interest, whichare visually captured by pixels of the pixel panel.

In one embodiment, the aspect of the segmented image that is to be thesubject of the histogram may include defining the source of thehistogram as the approximate boundary of the segmented image. Thesampling parameters also define the individual histogram values that areto be derived from the defined aspect of the segmented image. Thehistogram values may correspond to any characteristic of a pixelrepresentation of the segmented image. For example, the histogram valuesmay be defined, for pixels that are randomly selected using the samplingparameters, as the space between two pixels, the difference in pixelintensity between the two pixels, the respective lengths of orthanormallines that extend between two or more pairs of selected pixels.

Thus, in step 420, the histogram values are determined for randomlyselected pixels based on the sampling parameters. For example, if thesampling parameters define a boundary of the segmented image, then thehistogram values that are to be determined may correspond to thedistance between two randomly selected points on the boundary of thesegmented image.

The histogram values are collected a sufficient number of times so thatthe collection of histogram values are more invariant to the set oflocations from where the information is extracted. The collectedinformation is represented by its histogram. Depending of the source ofthe information, the histogram representation can be invariant torotation, translation and scale. Thus, in a given duration, the movement(including rotation and translation) of a body part may be representedby a histogram, and the representation is not compromised or affected bythe movement.

15 The histogram information to be collected may have many forms. Onealternative is the use of randomly chosen perpendicular lines as thesource for selecting pixels and determining histogram values. The lengthand/or the angle of each one of the perpendicular lines can be kept aspart of the histogram values. The perpendicular lines may extend betweenpixels randomly selected from, for example, a boundary of the segmentedimage, although the points forming portions or end-points of theperpendicular lines could be positioned anywhere on the object. In thisexample, if histogram values are taken for a sufficient number of linepairs, the statistics of the histogram are substantially invariantagainst the choice of the points.

Regardless of the source of the information, a histogram representationof an image may be meaningful and accurate, and invariant to the randomset of points, if the histogram values are collected a sufficient numberof times from pixels that are randomly selected. The invariance of thehistogram (to rotation, translation or scale) depends on the form ofrandom information used to construct the histogram. For example, if thedistance measured between randomly selected pixels of the boundary ofthe segmented image is used to construct the histogram, then the resultsprovided by the histogram are substantially invariant to rotation andtranslation but not to the scale of the object. Moreover, any lengthrelated random information source results in histograms that are notinvariant to scale. If invariance to scale is desired, then thecollection of the distance measures should be normalized with respect tothe maximum of this collection. As another example, a histogramcomprising a collection of angle information for discrete locations ofthe body part results in invariance to translation and scale, yet not ininvariance to rotation.

In addition, due to the nature of the histogram, a small variation onthe image results in a small variation in the histogram. This provides asignificant advantage for the application of histograms to shaperecognition applications, because small variations in the image do notsignificantly impact the image's representation with the histogram.

In many cases, the source for random information within the histogramcomes in forms of vectors, i.e. collection of numbers. For example, ifthe distance between points and the angle of the line are both kept,than each instance of the random source provides a two-dimensional data.In this case, the histogram of multidimensional data is to be obtained.A method for obtaining multidimensional histograms may be implemented.One such method uses a “k-means algorithm” to cluster the data and toobtain cluster centers. The cluster centers are used as the histogrambin centers. By counting each instance of the random information, i.e.each vector, towards the bins, the multidimensional histogram isobtained.

The resulting histogram, either one dimensional or multidimensional, isa representative (signature) of the underlying image of the hand shape.Step 440 provides that the histogram is stored as the representation fora particular body shape. In this way, the histogram may serve as asignature for the shape of the body part of interest.

FIGS. 5A-5I are a series of snap-shots that illustrate an image of abody part (a hand provided in column 510), the transition of a segmenteddepth image of the hand (provided in second column 520) over a timeperiod where the hand is gesturing, to an aspect of the image (theboundary) that is designated to provide the statistical information forthe histogram (provided in third column 530), to a histogram (fourthcolumn 540). The histogram in the fourth column 540 results in being thesignature for a shape of a particular hand shape. In the exampleprovided, random histogram of the distance information provides arepresentation that is invariant to rotation and translation of theshape. In addition, the distances are normalized with respect to themaximum observed distance. In this way, invariance to scale is achieved,so that hands varying in size have similar histograms for the samegesture. In addition, the histograms are resilient to small variationsin the shape. This allows the system to be invariant to small variationsin the image when the same posture is intended.

As shown in FIGS. 5A-5I, various hand gestures are each associated witha unique histogram. For example, a raised fist in FIG. 5C has adifferent histogram than two fingers extended in FIG. 5A. In oneembodiment, each static hand gesture captured in column 510 may beclassified as an input. In another embodiment, the hand gesturescaptured in column 510 represent a hand that is moving, and thehistogram is of just one instance while the hand is moving. In stillanother embodiment, all of the hand gestures collectively can beclassified as one set of input.

In another embodiment of the system, other methods in the literaturecould be used for shape representation. Among these methods are momentinvariants and descriptors, Fourier descriptors and elliptic Fourierdescriptors.

The pose of a body part may be represented by the position andorientation of the body part. The position of the body part may beprovided by the position of a designated reference position on that bodypart. FIG. 6 provides a more detailed description of a method fordetermining a pose of a body part of interest, where the body part ofinterest is assumed to be a hand.

In step 610, the position of a hand may be given by its center location.The center location may be based on the average of the coordinates ofevery point in the hand.

Step 620 provides that the normalized coordinate of each point (or asubstantial number of points) on the hand is obtained. The normalizedcoordinates may be obtained by subtracting the center locationcoordinate from each one of the coordinates.

In step 630, a singular value decomposition algorithm is applied to thenormalized coordinate of each point on the hand. This may require firstlisting the normalized coordinates into one column of a matrix. When thesingular decomposition algorithm is applied on the matrix, singularvectors result that define a shape of an ellipse. The resulting singularvectors correspond to the major and minor axis directions of theorientation of the hand. Similarly, the singular values given by themajor and minor axes correspond to lengths of the ellipse that best fitsthe hand. Thus, the pose can be defined by the direction and lengths ofthe major and minor axes that are identified for the hand.

The combination of the hand shape and pose give the body part posturerepresentation at a particular time. As will be described, additionalbody part representations may be derived in succeeding intervals inorder to determine a dynamic gesture representation.

F. Dynamic Gesture Representation

Embodiments of the invention provide for recognizing dynamic gesturerepresentation of a body part. Dynamic gesture representation may occurwhen the body part is moved, and the movement is to be recognized aspart of the overall gesture. In one embodiment, the dynamic gesturerepresentation corresponds to a series of body posture representations,taken over a duration of time.

10 For purpose of illustration, the description of the methods andtechniques in this section are for hand gestures. For example, a userdirecting one hand at an electronic device while raising two fingers maybe recognized and interpreted as a first command. In the same example,the user may make the same motion, but have his hand postured into afirst. This may be recognized as a completely different command. Theuser may enter one or more commands by taking his hand from a first totwo fingers extended outright. Still further, the direction of movementof the hand, with the posture of the fingers or palm (the posture couldalso be changing) may be recognizable as input. For example, a userpointing at a device while moving his hand towards it may be interpreteddifferent than a user pointing at the device while moving his hand awayfrom the device.

One of the challenges for a hand (or other body part) gesturerepresentation is the determination of the beginning and end of adynamic or moving hand gesture. Classification based algorithms, such asHidden Markov Models exist for this task, yet a robust solution wouldrequire a well-defined procedure.

Another approach is to use a delimiter functions. For example, one ormore of the following delimiter functions may be employed: (a) aspecific hand gesture to delimit the beginning and/or end of a handgesture; (b) a person stopping at the end of a gesture that is to berecognized; (c) a person creating a specific sound to delimit thebeginning and/or end of a hand gesture; and (d) for computerapplications and the like, the user inputting a specific key to delimitthe beginning and/or end of the hand gesture. Between the delimiterfunctions, the user may perform the dynamic gesture.

FIG. 7 illustrates another embodiment where the beginning and end of adynamic gesture are determined through analysis of discrete bodygestures recognized at consecutive instances of time. Thus, thebeginning and end of the gesture are not determined independently of therecognition analysis, but rather require the recognition analysis to beperformed. FIG. 7 illustrates one embodiment where statistical methodsare used to recognize the dynamic gestures.

In step 710, a posture representation vector for a particular instanceis constructed. The particular instance may be one of several instancesthat are determined to be part of the duration when the gesture takesplace. The posture representation vector may be constructed as a featurevector from the histograms used in determining the shape of the bodypart of interest and/or its pose. The feature vector corresponds to aone-dimensional matrix that comprises the bin values of the histogram,listed sequentially in the one dimension of the matrix. In anembodiment, the posture representation vector also includes separatevalues representing the location and the pose of the body at the giveninstance. The separate values may form the last few elements of thefeature vector.

Step 720 provides that the posture representation vector at two or moreconsecutive instances are compared to determine the beginning and end ofthe gesture. The basis of comparison may correspond to some designatedcriteria, or a user-specified criteria. When two or more consecutiveposture representation vectors are compared to be equal or close (basedon the designated comparison criteria), the conclusion may be drawn thatthe gesture was either stopped, or became stopped. Thus, for example,two consecutive posture representation vectors that are determined to beequal, followed by a different posture representation vector at the nextinstant, may represent the beginning of the gesture. Likely, when twoconsecutive posture representation vectors suddenly are equal, theindication is that a gesture has been completed.

Step 730 provides that a trajectory of posture representation vectorsare obtained. The trajectory tracks the posture representation vectorsfor a given duration that includes the duration of the gesture.

In step 740, the timeline between the beginning and end of the gestureis divided into intervals of approximately equal amounts of bodymovement. The amount of the body movement can be modeled by the changein the posture vector between the consecutive frames. As a consequence,the trajectory of the posture vectors between the beginning and end ofthe gesture is divided into equal-length (or equal time) intervals.Given where d is the total trajectory length, and k intervals areconstructed, each interval can be assumed to have a length ofapproximately d/k.

Once the intervals are determined, a representation of each interval isdetermined in step 750. The representation of each interval may berepresented through several techniques. For example, an average of theposture representation vectors in the interval can be used for therepresentation of that interval. Such representations may be intervalrepresentation vectors.

Step 760 provides that a gesture representation vector is constructedfrom the representation of the individual interval vectors. For example,the k interval representation vectors may be stacked into a gesturerepresentation vector.

In step 770, a gesture representation vector may be classified as a typeof gesture as described in Section G. The type of gesture may correlateto an input for a related electronic device.

In other embodiments, velocity and acceleration based features may beadded into the gesture representation vector. These would involve theaverage and/or instantaneous velocity/acceleration during the gesture.This is useful information since the velocity of a gesture as a methodof modifying the meaning of the gesture, say, emphasizing an action.(Such gestures may be deemed compound or combined.)

G. Classification

The final stage of the system is the classification of the gesturevectors into one of the specified gestures. In one embodiment,statistical classification may be used to perform this task. There arevarious statistical classification algorithms. Such classificationalgorithms include support vector machines, neural networks, nearestneighbor classification, or Hidden Markov Models. In general,statistical classification involves two main stages, a training stage(off-line) and a classification stage (on-line).

The training stage occurs off-line before the system is put into action.It involves a classifier-learning algorithm that takes a training set asinput and produces a classifier as its output. For embodiments such asdescribed herein, a training set is a collection of gesture vectors fora body part of interest. The collection of body gestures may be manuallylabeled. A classifier is a function that, for any new candidate gesture,tells what type of gesture it is. In other words, theclassifier-learning algorithm partitions the space of gesture vectorssuch that each partition corresponds to one of the gestures in thetraining set. The classifier function determines which partition agesture vector falls in.

The classification stage involves the application of the learntclassifier function on new cases. This stage occurs on-line anddetermines the decision of the system.

H. Communication Protocol Between User and Recognition System

According to one embodiment, a robust gesture recognition system isprovided that utilizes high level reasoning to confirm some or allimage-based classification decisions. In one embodiment, this is managedby a communication protocol that is defined between the user and thesystem. This protocol consists of a serious of confirmations from theuser regarding the system's decision. Once the user applies thedelimiter function, the system gives the most possible answer to theuser. The user accepts or rejects the decision by one of the following:(a) a hand gesture, (b) creating a specific sound, and (c) inputting atext or key to the computer.

If the system's first guess is incorrect, the system reveals the secondmost possible answer, and asks for the confirmation. This procedurecontinues for a couple of iterations, or until the system gives thecorrect decision. If the system cannot correctly guess the gesture in afew trials, it asks the user to redo the gesture.

In this manner, the user confirms the system's decision, and the robustoperation is attained. In addition, the user can undo his gesture bysimply rejecting the system's decision.

I. Description of System Operation

An example of how a system for implementing methods such as describedabove is provided in detail. Reference is made to elements of FIG. 1 forpurpose of facilitating the description. The sensor module 110 sensesthe presence of a hand, fingers, toes, or other natural elements withinregion 155, which corresponds to the sensing. The body posture module130, in combination with the sensor module 110, locates the position(along three axes X, Y and Z) and orientation (pitch, yaw, roll) ofuser's natural features (fingers, fingernails, hands, etc.). The systemmay collect the position and orientation information and describes theobjects shown in a digital (electronic) representation such as a datastructure, table of information, or series of object descriptors.

If there are multiple gestures (by hands or fingers) in the field ofview, the system also collects this information because it is computingthe distance to every pixel in the field of view. The distanceinformation for every object in view is simultaneously calculated, inreal-time. The body gesture representation 150 also interprets thedistance information to translate the natural user movement into aspecific gesture or meaning. A communication protocol such as describedbelow between the user and the system might be applied to confirm thesystem's understanding.

Once the system has processed the distance data to either recognize acertain action described in the system (e.g. a keypress or “thumbs-up”gesture) or describe the field of view in a general way (e.g. “largepainting motion in a nearby area”) it then creates an appropriateinterface event within the electronic system in a way that theelectronic system can understand. For instance, in the case of a virtualkeyboard such as described in U.S. provisional patent application No.60/163,445, priority to which was claimed for what is now U.S. Pat. No.6,614,422, (incorporated in its entirety herein for all purposes), thelocation of a keypress is correlated to the layout of a keyboard todetermine which key is pressed. The key-code for that key is then sentto the host system in a way that the host system can understand. Forinstance, in a personal computer, the “ASCII” value for that particularkey may be sent to the keyboard device driver. The system workscomparably for three dimensional objects. For instance, if “thumbs-up”were meant to turn up a gesture-controlled radio, then the appropriate“volume up” command would be sent to the electronic system. In thegeneral case, if a “large painting motion in a nearby area” were made, adata structure describing that event would be sent to a paint programthat would understand the data structure and cause the appropriate paintstroke to be recorded. (It is understood that a system able to acquirethree-dimensional positional information such as described inabove-referenced U.S. Pat. No. 6,614,422 is useable in implementing anembodiment such as shown in FIG. 1.)

Embodiments of the invention may support the use of multiple actionssimultaneously, and support interface events that are both known intoday's systems (e.g. keystroke, mouse movement) and those that are notknown by today's systems. For instance, if two fingers touched a surfaceat the same time, meaning that two keys were pressed at the same time,the system uses the distance information to interpret these eventsseparately and simultaneously so that two or more actions may berecorded at any given time. This is in contrast to existing systems,such as touch screens based on capacitive differences, which only enablethe user to complete one action (e.g. one “touch”) at any given time.Similarly, the system supports multiple gestures, which may not be knownin today's systems, simultaneously.

Once the event is known to the electronic system, (whether it be aspecific event like “G key has been pressed” or “Radio volume up” orgeneral “large painting motion in nearby area”) the application softwarerunning in the electronic system then reacts to the event. In the caseof the keyboard, this would affect a keystroke in the system, causing aletter to appear on a screen, for instance. In the case of thegesture-controlled radio, the volume would actually increase. In thegeneral case of the paint program, a painting stroke that approximatesthe size, intensity, and position of the motion would occur.

J. Application Examples

Various applications exist for a gesture or activity recognition systemsuch as described herein. For example, one application for an embodimentof the invention includes a video game input device that is able tointerpret finger, hand, and body motions into actions within the game.For instance, the game player may direct a punch in the air and the gameinterprets his motions into the game. Similarly, the user can interactwith a virtual reality system using his gestures.

Another application is for automobile control device that enables thedriver to control, with gestures, the functionality of one or moresystems within the car (e.g. thumb up to turn radio up, point down toturn air conditioner down, point at windows and roll the window up ordown using a gesture).

Another application is for identifying the locus of where a directionalmicrophone or camera needs to point. For the microphone application,identifying the direction where the sound comes from enables themicrophone to only pick up the sound from a narrow angle thuseliminating the background chatter.

Another application includes using a gesture recognition system such asdescribed to implement robotic controls that enable the user to controla “Waldo” or electronic arm by replicating the motion of his own limbs.Similarly, the user can communicate by a robot by using his gestures.

Another application for an embodiment of the invention includes a systemto interact with models of molecular or other tiny structures by usinggestures as if the tiny structures were of a larger physical size.

Still further, an embodiment provides for a computer having auser-interface that does not require any type of form-factor. Forinstance, the user can put in text input to a computer by using his/hergestures. The user can draw a number or character in the air, which isused as the input to a device.

Another application that can illustrate the diversity of embodimentsdescribed herein includes an electronic system for painting. In thiscase, the user is able to perform the following create a work of artusing the natural gestures of his/her own body, without the need for anytype of artistic tool. Thus, if an artist wishes to get the effect of“smushing” paint into the painting surface, the artist could, throughimplementation of an embodiment described herein, use his hand to“smush” in mid-air in order to control and produce the desired effect.

As another example, embodiments such as described enable an artist toemploy multiple artistic controls (e.g. two hands) simultaneously tocreate an improved artistic effect. For instance, with an embodimentsuch as described, the artist could “paint” a line with one hand andcontrol the change of color with the other so the exact desired colorgradation is achieved.

The artist may create the effect of controlling (changing) brushthickness by changing the profile of the contact point. For instance,the artist would use a fingernail for drawing a thin line by moving thefinger in the air or a thick line by moving the whole hand down. Theartist may create or edit a sophisticated work of art if the artist'slimbs or motor skills are impaired making it impossible to handle anartistic implement of any kind. In addition, photographs or realisticimages may be edited using finger, hand, or body controls in order toachieve a greater sense of control or artistic expression.

Still further, a work of art may be edit with a level of control thatdoes not exist in the world today. For example, consider its use in asystem that edits photographs. If the photo editor is looking at a womanin a photo and wanted to rustle her hair so that it appeared more messy,then the editor could just take his hand and move it in mid-air as if torustle someone's hair with the exact intensity and orientation hedesired. Then, the electronic photo system outfitted with this inventionwould interpret that motion and rustle the woman's hair appropriatelyaccording to the photo-editor's action. This provides great benefit overknown systems for editing photographs (e.g. pens, mouse, electronicpaintbrushes). In addition, the intensity of an artistic stroke may bemeasured to a greater degree and reflect the intensity in the work ofart. For instance, if the artist wants to archive a very bold stroke,the artist may, with embodiments such as described herein, strike hishand hard and quickly against the “virtual” painting area.

Further artistic applications include finger painting may be simulatedwithout any mess, and assisting handicapped artists that cannot hold oruse traditional artistic implements.

The application examples pertaining to art illustrate the diverse rangeof functionality and benefits that can be attained with embodiments suchas described by this application. Thus, while examples are given for thecase of an electronic system which allows for painting and art, numerousother functions and uses may be provided for by other embodiments of theinvention.

Embodiments of the invention may also detect and interpret gestures forpurpose of changing a parameter of a companion electronic device. Forexample, embodiments of the invention could be used as a remote controlfor a television, or as a wireless input mechanism for small devicessuch as cell phones and personal digital assistants.

For applications such as described, it might be desirable thattwo-handed gestures are used as input. In these cases, two differentpaths could be pursued. The first path involves the recognition of twohand gestures that work independently (turning two independent knobs).This would involve the application of the same recognition system oneach hand. As an alternative, the meaning of one hand may influence themeaning of the other.

K. Conclusion

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the invention. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

Thus, modifications and variations may be made to the disclosedembodiments without departing from the subject and spirit of the presentinvention as defined by the following claims.

1. A method enabling user-interaction with an electronic device using atleast one gesture made by a movable body part of said user, the methodincluding the following steps: (a) using a sensor system, operableabsent ambient light, from which three-dimensional coordinateinformation is obtainable to acquire three-dimensional space positionalinformation for a plurality of discrete regions on said body partrelative to said sensor system; (b) detecting from saidthree-dimensional positional information acquired at step (a) occurrenceof said at least one gesture; and (c) classifying each said gesturedetected at step (b) as at least one input to said electronic device. 2.The method of claim 1, wherein at step (a), said sensor system isprovided with an array of pixels to capture an image of a sceneincluding said at least one body part, wherein each pixel in said arraycorresponds to a discrete region of said scene.
 3. (canceled)
 4. Themethod of claim 1, wherein at least one said gesture is classified asone of (i) a dynamic gesture, and (ii) a static gesture.
 5. The methodof claim 1, wherein: at least a first gesture and a second gesture occursimultaneously; step (b) detects each said gesture; and step (c)classifies each said gesture; wherein classification at step (c)includes at least one of (i) classifying said first gesture and saidsecond gesture so as to comprise a single input to said device, and (ii)classifying said first gesture and said second gesture so as to comprisea first input and a second input to said device.
 6. The method of claim5, wherein said first gesture is made with one of a hand and a foot ofsaid user, and said second gesture is made with one of another hand andanother foot of said user
 7. The method of claim 1, wherein said atleast one gesture is compound and includes at least one of (i) avelocity component, (ii) an acceleration component, (iii) a directionalcomponent relative to location of said three-dimensional sensor system,and (iv) a gesture comprising movement.
 8. The method of claim 1,further including using sound detection to classify at least one saidgesture.
 9. The method of claim 2, further including segmenting saidbody part from a remainder of said scene, wherein segmenting includes atleast one of (i) using positional information to cluster together pixelsrepresenting portions of said body part, (ii) using positionalinformation to cluster together pixels representing portions of saidbody part based on similar depth information, and (iii) clusteringtogether pixels representing portions of said body part based at leastpartially on a depth of a discrete region from said system.
 10. Themethod of claim 1, further comprising identifying at least one of (i) ashape of said body part, (ii) a pose orientation of said body part,(iii) a shape of said body part identified by performing statisticalanalysis on said image, (iv) a pose of said body part, and (iv) a poseand orientation of said body part.
 11. The method of claim 2, furtherincluding performing a statistical analysis on said image, wherein dataanalysis includes at least one of (I) identifying a shape of said bodypart, and (II) performing a statistical analysis on an image portionincluding said body part segmented from said scene; wherein performing astatistical analysis includes at least one of (i) creating a histogramusing coordinates that identify pixels in said array of pixels, and (ii)creating a histogram using coordinates that identity pixels in saidarray of pixels on a boundary of said image.
 12. The method of claim 1,wherein step (c) includes classifying said gesture based on at least oneof (i) shape of said body part, (ii) pose of said body part, (iii)classifying a series of body postures occurring at multiple instances ofsaid time duration interval wherein a combination of said body posturesforms said gesture, (iv) using positional information of said body partto recognize said body part is making a gesture to the exclusion ofother body parts, (iv) using velocity information of said body part torecognize said body part is making a gesture to the exclusion of otherbody parts, and (v) using acceleration information of said body part torecognize said body part is making a gesture to the exclusion of otherbody parts.
 13. The method of claim 1, wherein said body part includesat least one of said person's hands, and step (c) includes using saidpositional information to classify a gesture created by at least one ofsaid user's hands.
 14. The method of claim 1, wherein step (c) includesusing positional information to classify a gesture formed by at leastone of a finger, a hand, a hand-held object, an arm, a shoulder, an eyeball, an eye lid, a head, a foot, and a leg of said user.
 15. The methodof claim 1, wherein step (c) classifies said gesture as input for anelectronic device selected from a group consisting of a computer, aportable computer, a keyboard, a television system, an audio system, agame console, a mobile phone, a robot, and an appliance.
 16. The methodof claim 1, wherein said at least one gesture is a dynamic gesture, anddetecting at step (b) includes at least one of (i) detecting anoccurrence of a first delimiter action signifying when said dynamicgesture starts, (ii) detecting an occurrence of a second delimiteraction signifying when said dynamic gesture ends, and (iii) andrecognizing that at least one of said first delimiter action and saidsecond delimiter action correspond to a specific posture of a body partof said user.
 17. The method of claim 1, wherein at detecting at step(b) includes detecting an occurrence of a delimiter action that createsa designated audible sound.
 18. The method of claim 1, further includingat least one of (i) communicating to said user gesture classificationbefore step (c) classifies said gesture as said input.
 19. The method ofclaim 17, further including detecting received confirmation from saiduser that said input is what said user intended to create.
 20. Themethod of claim 18, wherein detecting said received confirmationincludes detecting at least one of (i) another gesture by said user,(ii) an audible signal created by said user, and (iii) a manual entry bysaid user into said device.
 21. The method of claim 1, wherein at leastone of step (a) and step (c) is provided as instructions on acomputer-readable medium, said instructions being executable by at leastone processor to perform such step.